Transfer Learning for Video Recognition with Scarce Training Data for Deep Convolutional Neural Network

摘要

Unconstrained video recognition and Deep Convolution Network (DCN) are twoactive topics in computer vision recently. In this work, we apply DCNs asframe-based recognizers for video recognition. Our preliminary studies,however, show that video corpora with complete ground truth are usually notlarge and diverse enough to learn a robust model. The networks trained directlyon the video data set suffer from significant overfitting and have poorrecognition rate on the test set. The same lack-of-training-sample problemlimits the usage of deep models on a wide range of computer vision problemswhere obtaining training data are difficult. To overcome the problem, weperform transfer learning from images to videos to utilize the knowledge in theweakly labeled image corpus for video recognition. The image corpus help tolearn important visual patterns for natural images, while these patterns areignored by models trained only on the video corpus. Therefore, the resultantnetworks have better generalizability and better recognition rate. We show thatby means of transfer learning from image to video, we can learn a frame-basedrecognizer with only 4k videos. Because the image corpus is weakly labeled, theentire learning process requires only 4k annotated instances, which is far lessthan the million scale image data sets required by previous works. The sameapproach may be applied to other visual recognition tasks where only scarcetraining data is available, and it improves the applicability of DCNs invarious computer vision problems. Our experiments also reveal the correlationbetween meta-parameters and the performance of DCNs, given the properties ofthe target problem and data. These results lead to a heuristic formeta-parameter selection for future researches, which does not rely on the timeconsuming meta-parameter search.